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Introduction 


In a previous report (Layton, Smith and McCoy, 1993), an empirical study of 30 pilots using the 
Flight Planning Testbed was reported. This report describes an identical experiment using the 
Flight Planning Testbed (FPT), except that 27 airline dispatchers were studied. 

Five general questions were addressed in this study: 

1. Under what circumstances does the introduction of computer-generated suggestions 
(flight plans) influence the planning behavior of dispatchers (either in a beneficial or 
adverse manner)? 

2. What is the nature of such influences (i.e., how are the person’s cognitive 
processes changed)? 

3. How beneficial are the general design concepts underlying FPT (use of a 
graphical interface, embedding graphics in a spreadsheet, etc.)? 

4. How effective are the specific implementation decisions made in realizing 
these general design concepts? 

5. How effectively do dispatchers evaluate situations requiring replanning, and 
how effectively do they identify appropriate solutions to these situations? 

Below, we describe the design features of FPT, the methods used in this empirical study and our 
new findings. Because this is a replication of the previous study with 30 pilots (except for the fact 
that dispatchers were used as subjects), readers familiar with the previous report may want to skip 
the following sections on “The Context”, “The Flight Planning Testbed - Design Features”, and 
“Methods”, and begin reading the section on “Results and Discussion.” 
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The Context 


Enroute flight planning (Cohen, Leddo and Tolcott, 1989; Johannsen and Rouse, 1983; Rudolf, 
Homokoi and Sexton, 1990; Sorenson, Waters and Patmore, 1983) involves the modification of 
the aircraft route of flight (flight plan) of an airborne aircraft in response to problems with weather, 
air traffic, medical emergencies, mechanical failures, etc. The flight crew, air traffic controllers 
and airline company dispatchers all play important roles in this planning process. 

The flight plan stipulates what altitude and heading the plane will fly during various phases of the 
flight and what route the plane will take. The route in turn leads to the weather and air traffic that 
will be encountered along the way, affecting speed, safety, fuel efficiency, passenger comfort and 
arrival time. 

The planner, then, is concerned with getting from a given origin to a given destination in a timely 
and cost-effective fashion, while maintaining flight safety and passenger comfort The planner 
must consider what routes to take (these routes consist of navigational fixes and jet routes, the so- 
called ‘highways in the sky’ that connect the navigational fixes), what altitudes to fly, what 
weather to avoid (including winds, thunderstorms, freezing rain, and turbulence), and he/she must 
consider the ever changing characteristics of the plane (for example, the weight of the plane 
decreases as more fuel is consumed). 

The initial flight plan is rarely followed exactly, due to unforeseen events occurring while enroute. 
Indeed, minor changes in flight plans are frequently made and major changes are common. 
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These amendments to the original plan are due to the dynamic, unpredictable nature of the “world” 
in which the plans are carried out. Weather patterns do not always develop as predicted, resulting 
in unexpected areas of turbulence, less favorable tail winds or storms that must be avoided. Air 
traffic congestion may delay take-off or restrict the plane to lower than planned altitudes. Airport 
or runway closures can cause major disruptions, not just for one aircraft, but for everyone 
planning on landing at that airport Mechanical failures, medical emergencies or other critical 
problems may force the plane to divert to an unplanned airport 

In short enroute flight planning is very large and complex problem. Multiple goals must be 
considered in a highly stochastic environment where multiple plans must be coordinated (Hayes- 
Roth and Hayes-Roth, 1979; Hoc, 1988; Miller, Galanter and Pribram, 1960; Sacerdoti, 1974; 
Schank and Abelson, 1977; Stefik, 1981; Suchman, 1987; Wilensky, 1983). 

The Flight Planning Testbed - Design Features 

The Flight Planning Testbed (FPT) was developed to test several cooperative planning system 
design concepts (Coombs and Alty, 1987; Lehner and Zirk, 1987; Shute and Smith, 1993; 
Thierauf, 1988). This design was developed following an extensive cognitive task analysis 
(Smith, McCoy, Layton, and Bihari, 1992). The basic flight planning system performs a number 
of functions in response to input from a human operator. The system allows the user (either a pilot 
or a dispatcher) to develop and display up to four flight plans in conjunction with weather 
information and to obtain feedback in terms of flight parameters such as fuel, time, and distance. 
The weather information consists of both graphic depictions and verbal descriptions and can be 
displayed at several altitudes. The displays show the entire flight path, thus emphasizing global 
solutions to problems. In addition, the person can manipulate the display time to see the 
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relationship between the weather information and the plane's position. The system computes the 
optimal altitude profile to minimize fuel consumption, arrival times at navigational fixes, and fuel 
remaining at those fixes, based on wind components. It will also determine these flight parameters 
given a user-selected altitude profile. 

The basic system runs on a Macintosh Ilfx with two color monitors. The features and functions on 
each monitor are discussed in turn below. 

Left Monitor 

The displays and controls on the left monitor are shown in Figure 1 . (In all of the figures which 
depict system displays, some of the information loses saliency as printed here in black and white 
instead of color.) 


Insert Figure 1 about here 


The primary feature on the left monitor is a map display. This display depicts the continental 
United States, the aircraft position, and planned routes. Several pieces of information can be 
overlaid on this map. This information includes: 

1 . Weather information, with overlays of composite cloud and composite radar charts, fronts, 
and cloud cover, radar and winds at specific altitudes. 

2. Navigational fixes and jet routes. (See Figure 2.) 

All weather information is available for two display times, the ‘current’ time and a one hour 
forecast When a forecast is displayed, the aircraft is displayed in its predicted position (on each 
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route) at the forecast time, as well as at its current position. The user can also ‘zoom in’ on a region 
of the map, which replaces that map of the continental U.S. with a magnification of an area 
surrounding an operator- selected point 


Insert Figure 2 about here 


Right Monitor 

The right monitor displays and controls are shown in Figure 3. It displays a ‘flight log’ of a route. 
This flight log is essentially a spreadsheet which depicts each segment of the route (i.e., all of the 
navigational fixes and jet routes which make up the route), as well as information pertinent to those 
segments. The flight log also graphically displays the planned altitudes for the route and the least- 
fuel-consumption altitudes for that route. Finally, the flight log displays weather information 
which is pertinent to the route. For example, turbulence information is on by default, but the 
person can also select information on the winds. The turbulence information that is presented is a 
one-word summary of the maximum turbulence on a given flight segment The operator can get a 
more detailed description of that information (available ‘pilot reports’ or ‘pireps’) by selecting 
(‘clicking’ on) the one-word summary. 


Insert Figure 3 about here 


The other display on this monitor (at the bottom of the screen) shows important flight parameters 
for all four alternative routes upon arrival at the destination. These parameters include time of 
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arrival, time enroute, fuel remaining, and total distance. This display allows users to compare the 
‘bottom line’ for each route. 

FPT - Important Features 

The design principles underlying FPT as a cooperative planning system are discussed in detail in 
Smith, McCoy, Layton and Bihari (1992). Five of the most significant considerations, however, 
are: 

1 . Provide tools that allow cooperative planning at different levels of abstraction 
(inspired by the work of Sacerdotti, 1974; Hayes-Roth and Hayes-Roth, 1979; 
Shute and Smith, 1993; and Suchman, 1987); 

2 . Provide the human planner with data displays and representations to support plan 
generation and evaluation at these different levels of abstraction; 

3 . Provide interfaces to the available support tools that allow the person to easily 
communicate desired tradeoffs among goals; 

4. Provide tools that help the person predict the outcomes of various plans (Coombs 
and Alty, 1987); 

5 . Incorporate a graphical interface that allows the person to view and explore 
alternative plans in the context of the relevant data (i.e.,weather displays). 

Below we describe an empirical study to assess some of these design considerations. 

Methods 

In the study described below, FPT was used as a testbed to study the effects of different design 
features on cooperative problem solving performance. Briefly, each of the twenty seven subjects 
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(professional airline dispatchers) was asked to think aloud (Ericsson and Simon, 1984) as he used 
one of three alternative system designs. (Subjects were randomly assigned to the alternative system 
versions.) Each subject was trained on the use of that version of the system and given four cases 
to solve. 

System Designs 

As mentioned above, three different enroute flight planning support systems were designed. In 
actuality, these three systems represented variations on the levels and timing of support provided 
by the computer. These variations on the system design represented the independent variable 
studied in this experiment. 

The ‘Sketching Only’ System. The ‘sketching only’ system allowed the human planner to 
sketch proposed flight paths on a map display, while the computer filled in lower level details 
(such as fuel remaining, time of arrival, and recommended altitudes) using an optimization 
program. In this version, the person was responsible for proposing the alternate paths, while the 
computer was responsible for providing feedback on those solutions. 

The sketching of routes was carried out by displaying the jet routes and navigational fixes and 
selecting (‘clicking’ on) each navigational fix that the dispatcher wanted the airplane to pass 
through. This placed a slight restriction on planning because vectoring can normally be requested 
to fly direct routes from one point to another. However, this approach allowed the planner to 
develop general solutions with the understanding that these solutions were not necessarily the exact 
routes that would be flown. 
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The ‘Route Constraints and Sketching’ System. The ‘route constraints and sketching’ 
system retained all of the capabilities of the ‘sketching only’ system and it added another capability: 
The person could specify higher level constraints on the solution he desired and then ask the 
computer to find the shortest distance route which satisfied those constraints. The constraints that 
could be specified were the maximum allowable turbulence, the maximum allowable precipitation, 
and the destination. (It is easy to see how this interface design concept could be extended to 
include other constraints such as earliest and latest desired arrival times or percentage of passengers 
making their connections.) 

The user could specify constraints on the solution he desired from the computer. The computer 
would then recommend alternatives. In addition, the user had a means, through the sketching tool, 
of exploring specific routes himself. 

The ‘Automatic Route Constraints, Route Constraints, and Sketching’ System. 

The ‘automatic route constraints, route constraints, and sketching’ version took the computer’s 
involvement one step further in that the computer automatically suggested a deviation (based on 
default constraints of no turbulence, no precipitation, and the originally planned destination) as 
soon as it detected a problem with the original route. This form of tool is akin to an autonomous 
support system that automatically suggests solutions to detected problems. This system also made 
available the ‘route constraints’ tool of the previous system and the ‘sketching’ tool of the previous 
two systems. 

Underlying all three system designs is the incorporation of tools to support asking “what if’ 
questions. That is, these tools help the operator to ask “what if I do this?” (e.g., “What type of 
solution does the computer suggest if I use constraints of light turbulence and moderate 
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precipitation?”, or “What happens to my fuel remaining if I deviate north instead of south?”). We 
were interested in whether people used the tools available to them, how the available tools affected 
the cognitive processes of the person using the system, and how the available tools affected the 
solutions that person chose. 

Cases - Characteristics and General Predictions 

Following training on the use of the system, each of the subjects was presented four enroute flight 
planning cases in which he was given some preliminary information about the flight (e.g., origin, 
destination, time of day, etc.) and was then told to “decide what the plane should do”. All of the 
subjects went through the same four cases in the same order. Whereas the subjects in the 
‘sketching only’ and ‘route constraints and sketching’ conditions started each case with only their 
original route of flight, the subjects in the ‘automatic route constraints, route constraints, and 
sketching’ condition were also given an alternate route suggested by the computer based on the 
default constraints of Finding a route that was predicted to avoid all turbulence and precipitation. 

Cases 1-3 can all be characterized as having large “solution spaces”, that is, the number of 
plausible specific flight paths available to accomplish a particular deviation (such as going north of 
the storm) was very large. This characteristic was expected to put the subjects in the ‘sketching 
only’ version at a disadvantage in terms of finding fuel and time-efficient alternative routes. It was 
also expected to cause the ‘sketching only’ subjects to develop a larger number of specific flight 
plans for comparison. 

All four cases could be described as having a large “data space” in the sense that the types and 
amounts of data available in the different displays were fairly large (although still small by 
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comparison with the types and amounts currently being proposed for commercial systems at 
several of the major airlines). This characteristic was expected to be most important in Case 2 
(where a failure to note the headwinds to the south might lead to selection of the less efficient 
southern deviation by subjects sketching their own solutions) and in Case 3 (where a failure to 
look carefully at the location of the plane in relation to current and forecast weather might 
contribute to the acceptance of a poor plan). 

Case 3 had the further property that the limitations of the computer’s knowledge led to “brittle” 
performance, in which the computer generated a very poor suggestion for an alternative route. 
This brittleness was due to the fact that in searching for flight plans, the computer treated forecasts 
as reality. Uncertainty associated with the forecasts was not considered in the computer’s 
reasoning. This case was included to see whether subjects in the ‘automatic’ version would be 
more likely to be drawn into the computer’s “world”, consequently failing to use their own 
knowledge of the uncertainty associated with such a forecast to reject the computer’s 
recommendation. 

Finally Case 4, which has a much smaller “solution space,” is interesting because it introduces a 
conflict between a common heuristic used in selecting a flight amendment and the fuel and time 
efficiency of the alternative routes. Further details on these cases are included in discussions of 
the results. 


Results and Discussion 

A total of twenty-seven dispatchers from nine commercial airlines were studied. Ten used the 
‘sketching only’ version of FPT, 9 the ‘route constraints’ version, and 8 the ‘automatic’ version. 
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These dispatchers had from two to twenty-two years of experience as professional dispatchers, 
with an average of 9.2 years. There was no obvious relationship between any of the performance 
measures (reported in the following sections) and years of experience. 

Subjective Responses 

Since the subjects in this study were professional dispatchers, their reactions to the system 
represent valuable data concerning: 

1 . The potential usefulness of such a tool to aid them in their job; 

2. The value of the general design concepts underlying the system. 

In particular, these general design concepts include the use of a graphical interface for the 
generation of alternative flight plans, the availability of the route constraints function to control 
computer-generated flight plans, and the embedding of graphics in the spreadsheet to provide 
access to data on turbulence and winds along the planned route. 

‘Sketching Only’ Subjects. These ten subjects were unanimously enthusiastic about the 
value of such a system in general, and about the desirability of the graphical interface as a means to 
explore alternative flight amendments: 

“I love it. I like the idea of being able to see exactly where that route is on the screen right 
in front of me and see where the weather is in relationship to where that airplane is flying.” 

“This has the advantage of giving you a pictorial view of the alternate routes. I think it 
would be very useful. This would certainly make things a lot quicker and easier.” 
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This is a pretty good tool. It’s basically a what-if machine.” 


“This sure does speed up the process.” 

“This is going to improve the industry, bring it a long way.” 

“Right now at work what we have to do is visualize the routes and the fixes and compare 
them to a separate radar display on a different CRT. I think the overlay system on this 
system would be excellent.” 

“I wouldn’t mind having a system like that. It’s a nice system. It’s more visual.” 

“It’s nice to see the projected movement of the thunderstorms and how that affects your 
flight path in the future.” 

“You can compare on one screen all of the different plans.” 

“The turbulence reporting is nice.” 

“It’s not a hard program. It’s very easy to pick up on.” 

“At work we have to build a complete flight plan the way we do it This does it much 
quicker.” 
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‘Route Constraints’ Subjects. These 9 subjects were similarly positive about the system’s 
usefulness and underlying design concepts (including the route constraints function, which was 
unavailable to the ‘sketching only’ subjects): 

“It’s surprisingly sophisticated compared to some flight planning systems I’ve seen. 
Everything is tied together very nicely.” 

“I think this would be a tremendous improvement over the way that we do it now.” 

“This is fun. I wish we had this at work.” 

“This tool would be helpful in training.” 

“Boy, would I like to have one of these at work. This is incredible. Beautiful.” 

“It gives you an opportunity to check out what taking immediate action might do versus 
waiting until the last minute where you’d have to deviate more than originally planned.” 

“I like it because it lets you do a lot of what-ifs a lot quicker than you can on a normal 
map.” 

“I like the idea of being able to project the route of flight of an airplane.” 

“It’d be neat to have a tool like this to draw things out, have the airways right on the 
screen.” 
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‘Automatic’ Subjects. The 8 dispatchers studied with this version had similar responses: 

“This has got to be so complex to build. This is just absolutely phenomenal.” 

“Having the weather overlaid on jet routes is wonderful. We don’t have this yet” 

“This would be very useful.” 

“Since I work a lot of international flights, it’d be nice to have this kind of weather 
information available at my desk.” 

“I like having the route structure available right on the computer, being able to reroute them 
like that rather than, right now we take the maps out and start drawing points and then we 
type the route into the computer.” 

“If you presented the same scenarios to me with the system we have in the office we’d be 
here another couple of hours.” 

“It provides a very quick way of comparing fuel bums in circumstances where you would 
have multiple routes to select from. The fuel bum feature comes in really handy.” 

“This is neat I like this. Most people, they are crises managers. I like to have the attitude 
of to have a plan of what we’re gonna do if something happens as opposed to waiting for it 
to surprise me.” 
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“This was definitely much better than what I have at work right now. The ability to put 
weather over routes and the ability to put a forecast in and look ahead is much better than 
what I have.” 

Subjective Responses-Summary. The evaluation of FPT by dispatchers in their debriefs 
leaves little doubt that they perceive that the system would help them to perform tasks that are a 
normal part of their current jobs. It is also clear that they feel the underlying design concepts could 
provide a very helpful tool. 

Such responses, of course, do not tell us how the design features of FPT actually influence 
performance. The following objective data, however, provide answers to this question. 

Factors Influencing Route Selection. 

Given the nature of the data collected (concurrent verbal reports), it is impossible to identify all of 
the factors considered by a dispatcher in selecting a particular alternative flight plan in one of our 
scenarios (Ericsson and Simon, 1984). At various points in the transcripts, however, there is 
evidence of one or more factors being considered. 

Below is a composite list of all of the factors so identified. Such a list is potentially valuable to 
guide in further system design and evaluation and to help in developing dispatcher training and 
testing systems. Many of these factors are interrelated: 

1 . Fuel consumption as a cost; 

2. Fuel consumption as it relates to fuel reserves (a safety concern); 

3 . Arrival time as it relates to the published schedule and to passenger connections; 
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4 . Turbulence (current; predicted; cause; levels of uncertainty associate with the 
forecast); 

5. Thunderstorm activity (current; predicted; cause; levels of uncertainty associated 
with the forecast); 

6 . Passenger comfort; 

7 . The availability of alternatives or options to deal with an unexpected problem if it 
arises (including alternative routes to the planned destination and alternative 
destinations); 

8 . Characteristics of possible alternative destinations (weather; runway conditions; 
closings; air traffic activity; maintenance and support facilities); 

9. Characteristics of the planned destination (weather; runway conditions; closings; air 
traffic activity); 

10. Air traffic patterns enroute and on approach to the destination; 

1 1 . Preferred alternate routes by ATC; 

12. Approval (or the likelihood of approval) of a reroute by ATC; 

13. Expectations regarding the ability of the flight crew and ATC to detect and deal with 
minor problems on their own when they arise (without assistance from Dispatch); 

1 4. Expectations regarding the likely air traffic along various routes due to rerouting to 
avoid the same storm; 

1 5 . Winds aloft and their effect on fuel consumption and arrival time; 

1 6 . Availability of jet routes or vectors. 

This list of factors itself is not complete - there are clearly other factors that would be relevant to 
other scenarios and there are likely to be factors considered by dispatchers in our scenarios that 
simply were not verbalized. It does, however, serve to suggest the complexity of the flight 
planning task and indicates why it is important to think in terms of cooperative problem-solving 
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systems that support the dispatcher, rather than automation to replace the dispatcher. It is clear that 
the complexity of this task is beyond currently feasible methods for designing an autonomous 
computerized problem-solver. The list also serves to point out that flight planning involves 
cooperation between several parties (ATC, the flight crew and Dispatch). 

The Influence of System Design on Performance 

In our previous study of 30 pilots running in an otherwise identical experiment (Layton, Smith and 
McCoy, 1993), we found sizeable effects of system design (‘sketching only’ vs. ‘route 
constraints’ vs. ‘automatic’) on performance. In some situations, access to the computer-generated 
suggestions improved performance, while in others it impaired performance. Similar analyses are 
described below for this study of 27 dispatchers. 

Case 1 . The following scenario was read to the subjects prior to their working on this case: 

“It is summer and you’re on a flight from Battleground (Portland) to Northbrook. The 
dispatcher gave you a southerly route in order to avoid an occluded front The front has 
dropped to the south as well, however, and has generated some thunderstorms. Time out 
was 1700 Zulu and the plane is five minutes into the flight. Decide what you think the 
plane should do.” 

For subjects in the treatment condition in which the computer automatically suggested a solution 
upon loading the case, the following two lines were added (prior to “Decide what you think...”): 
“The computer has suggested the orange route as an alternative to the original plan (the 
green route) based on constraints of no turbulence and no precipitation. You may accept 
either of these plans or develop your own.” 
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The original route, (the middle route) the current aircraft position, and the current composite radar 
for Case 1 are shown in Figure 1 . The radar returns show a solid line of thunderstorms with cell 
tops at 37,000 feet (For this experiment, the dispatchers were told the aircraft’s maximum altitude 
was 33,000 feet.) Furthermore, the gap between the two cells was forecast to close. Therefore, a 
deviation was obviously required. The forecast storm movement was to the east, but was very 
small. 

Case 1 - Hypotheses. The previous study of 30 pilots highlighted three important results: 

1 . Because of the large “solution space” (i.e., the large number of possible routes), 
pilots in the ‘sketching only’ version were less likely to find the “best” route in 
terms of fuel consumption and time; 

2. Pilots using the ‘sketching only’ version of the system were influenced to more 
carefully consider the uncertainty associated with the storm and consequendy 
tended to choose a more conservative flight path; 

3 . In spite of the problems highlighted in the two points above, all 30 pilots found a 
solution that deviated to the north of the storm that was at least satisfactory in terms 
of all relevant factors. 

Below we contrast the performances of the 27 dispatchers in terms of these previous findings. To 
provide a concrete sense of the performances of the subjects, the behaviors of 3 representative 
dispatchers are first summarized. Then summary statistics are provided for the entire group. 

Case 1 - Sample Subjects. Below, the performances of three representative subjects (one from 
each system version) are described. 
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‘Sketching Only’ - Subject 24. This dispatcher first looked at the current composite radar display 
and asked himself: 

“Should I go north or should I go south.” 

He then looked at the current composite clouds display and the winds, checking the winds at 
different altitudes, commenting: 

“Looks like more tail winds to the south and less favorable winds to the north, but the 
shorter distance may make up for it.” 

He then looked at the current composite radar again, zooming in on the portion of the map showing 
the storm and superimposing the jet routes. He noted: 

“The earlier we start deviating the more mileage we’re going to save.” 

He sketched a route from BTG to MYL to HLA to BIL to DPR to RWF to ODI to DLL to BAE to 
OBK. (This route stays further north of the storm than the route recommended by the computer in 
the other two versions of the system.) 

He then scrolled through the spreadsheet for this new route, saying: 

“Now let me scroll for turbulence.” 

Checking the forecast for the composite radar, he commented: 

“We’re north of the weather and the pilot has the option as he gets closer that if he wants to 
deviate further north he can. If time was a factor, I’d go ahead and take this route now, but 
we still have plenty of time.” 

He then sketched another route going around the storm to the south, concluding: 
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“This uses quite a bit more fuel. About 3000 pounds more and much greater distance.” 

He therefore selected the route he had sketched to the north of the storm. 

‘Route Constraints’ - Subject 4. This dispatcher looked at the current composite clouds and 
composite radar displays, stating: 

“I’ll see what direction the weather’s moving and then go north or south of it.” 

He also checked the current winds and fronts and stated: 

“I’m gonna go to the south.” 

Before sketching a route to the south, however, he overlaid the jet routes and decided: 

“I’m gonna let the computer pick a route now just to expedite things.” 

He set constraints of no turbulence and no precipitation and let the computer find a route, noting: 
“The computer went north. I’m glad I did that. It saved me a lot of time if I’d done that 
myself by letting the computer pick first.” 

He looked at the spreadsheet for that route and commented: 

“Less time, more fuel, no turbulence, no precipitation.” 

He consequently immediately picked the computer-suggested northern deviation. (Note that he did 
not look at the forecast, and that there is no evidence that he thought about the uncertainty 
associated with the weather forecast.) 

‘Automatic’ - Subject 1 1. This dispatcher looked at the current displays for fronts and radar at 
different altitudes. He scrolled through the spreadsheet for the automatically suggested northern 
deviation, stating: 

“Orange route looks good as far as the weather.” 
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He displayed the jet routes and zoomed in, looking at the current and forecast weather displays for 
fronts and radar. 

“Doesn’t look like it’s moving too much. Only 10 miles an hour to the east Going to the 
south doesn’t look like it’ll save any time.” 

He then displayed the winds, noting (because of the tailwinds to the south): 

“I don’t know. It might Have to take a look down there.” 

He sketched a route to the south, but rejected it: 

“That takes 45 minutes longer and bums more fuel, so I’ll stick with the orange route.” 

Thus, he selected the computer-generated northern deviation. 

Case 1 - Route Selection. As in the previous study with 30 pilots, because of the large number of 
possible paths, some of the subjects failed to find the most fuel efficient point at which to begin 
deviating from the original route to the north. In the study of 30 pilots, 2 of the 10 pilots using the 
‘sketching only’ version failed to find this fuel efficient choice. In this study, 7 of the 10 
dispatchers using the ‘sketching only’ version similarly failed to find this fuel efficient choice. 

Dispatchers using the ‘sketching only’ version were, however, much more likely to consider the 
uncertainty in the weather forecast than dispatchers using the other two versions. This influence of 
the system version used was reflected in both the final route selected and the concurrent verbal 
reports. Six of 10 dispatchers using the ‘sketching only’ version selected a more conservative 
northern deviation (as did 6 of 10 pilots in the previous study). None of the 9 dispatchers using 
the ‘route constraints’ version selected a more conservative northern deviation than the computer’s 
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suggested northern deviation. Only one of the 8 subjects using the ‘automatic’ version picked a 
more conservative northern deviation. 

This difference in the tendency to closely consider the uncertainty associated with the forecast 
(leading to selection of a more conservative northern deviation) was further illustrated by the verbal 
reports. Subjects who selected the more conservative deviation made statements like: 

“By the time he gets there that might have moved in. Let’s stay north a little longer.” 

“I like this better. I’m willing to spend a little extra money to give him that cushion. If by 
chance the frontal system shifts a little, I’m giving him a little room to fire proof his buns.” 

Subjects who were influenced by the computer’s suggestion, however, typically made statements 
that failed to reveal any consideration of the uncertainty associated with the forecast: 

“All the info I have available suggests the computer has selected the best route, with not 
only no significant precipitation, no turbulence, but the time enroute, fuel bum and 
everything is better.” 

In spite of these differences in performance, though, 26 of the 27 dispatchers selected some 
northern deviation that was certainly satisfactory, and the one who selected a southern deviation 
also picked an acceptable route. 

Case 1 - Differences in Mental Models. There was one dispatcher who chose to deviate south even 
though he looked at a northern deviation. The most interesting insight provided by his data was 
the very significant difference in his model of the weather situation compared to the models of the 
dispatchers who deviated north: 
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“If you run him north into the low, I suspect you’d get turbulence aloft. If you run him 
south of the front, the possibility there is that the thunderstorms would extend 
southwestward as you try to route him around it. But the cold front in this case is small, so 
probably in this case I would elect to route the flight around the south end.” 

This assessment of the situation contrasts with that of the dispatchers who deviated north: 

“South. This is a crummy route.” 

“Given the season of the year, I don’t think its going to build to the north. I think it’s 
going to build in the south more.” 

“Going to the south side, the thing already has a history of sliding south, so you may run 
into the same problem a second time. You don’t want to do that.” 

Case 1 - Differences in Information Seeking. The study of 30 pilots found no significant 
differences in displays viewed across the different system versions for Case 1. This study found 
similar results, as shown in Table 1. The table indicates, for instance, that 10 of 10 subjects using 
the ‘sketching only’ version looked at the radar weather. 

Table 1. Information Viewed in Case 1 



Radar 

Fronts 

Winds 

Clouds 

Jet Routes 

‘Sketching Only’ 

10/10 

8/10 

6/10 

3/10 

10/10 

‘Route Constraints’ 

9/9 

9/9 

3/9 

4/9 

8/9 

‘Automatic’ 

8/8 

6/8 

7/8 

3/8 

8/8 
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Case 1 - Other Interesting Behaviors. As discussed earlier in general terms, there are a number of 
factors that should be considered in evaluating an alternative flight plan. It is interesting to note 
that only 2 of the 27 dispatchers showed any verbal evidence of considering one important factor, 
air traffic patterns: 

“I’m going to try to go around it to the north, mainly because of the traffic flows out of 
O’Hare.” 

“You’re bringing them into Chicago from the north. That’s usually good from the west 
side, rather than bringing them in from the south. They get a lot of traffic from the south.” 

In addition, it is interesting to note the way in which some dispatchers explicitly viewed the flight 
crew as a resource to detect problems and make modifications to a plan as necessary when new 
data becomes available: 

“I would remind the crew to contact me when he gets just in the vicinity of Billings to 
evaluate whether he needs to go a little further north.” 

Finally, two dispatchers described interactions with flight crews or ATC: 

“That’s actually where I’ve had arguments with pilots [about the desirability of spending “a 
little extra money to give him that cushion”]. They’re willing to do this [cut closer to the 
predicted storm activity]. When I tell them what it’ll cost I normally can get them to agree.” 

“The way it’s supposed to work, the captain, if he’s given a reroute by ATC or if he wants 
to go a different way, he’s supposed to call us and let us analyze it. If we notice, like in 
this case, like a line of thunderstorms along the planned route of flight, we’ll either get a 
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hold of them earlier or, yeah, that’s what we gotta do, we gotta get a hold of them as early 
as we can to try and change things around, let them warn ATC as far in advance as they 
can. A lot of times ATC will start doing it before we get a chance to get a hold of them 
‘cause they just start flowing stuff around too.” 

Case 1 - Summary. The results of this study for Case 1 were very consistent with those of the 
previous study with 30 pilots. First, it illustrated the potential value of the computer as a tool to 
identify fuel- and time-efficient routes around bad weather. Even with the graphical interface, 
dispatchers frequently had difficulty identifying the most fuel-efficient point at which to start 
deviating from the original route. 

Second, although the use of the computer to generate fuel-efficient deviations was beneficial to 
overall performance, there was again strong evidence that such computer-generated suggestions 
also had a potentially detrimental effect: 

Dispatchers who first viewed the computer’s suggested deviation were much less likely to 
consider the uncertainty associated with the forecast than those dispatchers who had to 
sketch their own routes. The graphical interface for sketching new routes tended to cause 
the dispatchers using the ‘sketching only’ version to look more closely at the weather, to 
consider the uncertainty associated with the forecast, and to consider its relationship to the 
alternative paths available. 

Nevertheless, in Case 1 all of the dispatchers generated plans that were at least satisfactory. 

Another interesting insight provided by the data from Case 1 was the very significant differences in 
the “mental models” that different dispatchers developed regarding the weather situation, and in the 
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different factors that the dispatchers considered (such as air traffic patterns). These individual 
differences have implications for the training of dispatchers. 

Finally, Case 1 provided hints regarding different “models” of cooperation among dispatchers, 
flight crews and ATC. The data indicate that, because of their different goals and roles, 
information sources, workloads and resources, all three parties play a part in the detection of 
problems and in identifying potential solutions. 

Case 2. Case 2 was designed so that there were two initially plausible directions for deviating 
(north or south of a storm). The scenario consisted of the following: 

"It's summer and the plane is eight minutes into a flight from Oakland to Joliet You got 
off the ground at 1600 Zulu. You notice that there is a solid line of convective 
thunderstorms directly in your path. Decide what you think the plane should do." 

Figure 5 shows the weather for this case. 

Case 2 - Hypotheses. Case 2 was designed so that the preferability of a northern or southern 
deviation around the storm was not immediately obvious. Consequently, the choice made was 
rather sensitive to the specific data viewed and the dispatcher’s mental model of the situation. 

In particular, our previous study of 30 pilots suggested: 

1 . Because of the moderately large “data space” (i.e., the fairly large number of 
choices of data to select from for viewing), some individuals are likely to miss 
important data (winds, forecast radar weather, or turbulence); 
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2 . There are likely to be significant differences in the mental models of the weather 
developed by different individuals; 

3 . Because of Points 1 and 2 above, there are likely to be disagreements in deciding 
whether to deviate north or south of the storm; 

4. Access to the computer’s suggestion (a northern deviation) is likely to significantly 
bias situation assessment, leading more people using the ‘route constraints’ and 
‘automatic’ versions to select a northern deviation. 

Case 2 - Sample Subjects. The performances of three representative subjects are described below. 

‘Sketching Only’ - Subject 26. This dispatcher looked at the current displays for composite clouds 
and, composite radar and looked at current radar and winds at different altitudes, commenting: 

“I guess that’s just summertime activity over the Rockies. Some people might want to try 
to thread through there but to me that’s not a good idea.” 

He then displayed the jet routes and turned the winds off to: 

“get rid of a little clutter.” 

He zoomed in on the storm, scrolled along the original route on the spreadsheet to look at 
turbulence and winds, and then sketched a route south of the storms, noting: 

“8000 pounds, which would be plenty.” 

He again scrolled along the spreadsheet looking at turbulence and winds, this time for his southern 
deviation, observing that there was a: 

“change from a headwind to a tailwind.” 
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He sketched another deviation, this time to the north of the storms, concluding: 


“We saved about 1000 pounds by coming the northern route than the southern route.” 

He looked at the current winds again and decided: 

“We shouldn’t have to worry about having it drift north into our route.” 

Note that he never looked at the weather forecast on the map. He consequently selected the 
northern deviation. 

‘Route Constraints’ - Subject 14. This subject began by looking at the current composite radar, 
composite clouds and fronts, scrolled along the spreadsheet looking at turbulence along the original 
route, and looked at the current and forecast radar at different altitudes, stating: 

“At this point I’d be leaning towards a reroute to the south.” 

He zoomed in, turned on the jet routes, and asked the computer to find a route with no turbulence 
or precipitation. He then scrolled along the spreadsheet or the computer-suggested northern 
deviation. 

At that point he moved back and forth between the current and forecast radar weather to: 

“take a look at the projected movement” 

He subsequendy sketched a southern deviation, rejecting it but commenting: 

“The southern route is using more fuel and time than the computer-projected route. The 
concern on this [the computer’s suggestion] would be that the route, that looks like the 
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weather is actually building to the north more than it is to the south, and before I’d select 
the northern route. I’d want to look at what the deviation would be if I had to go further 
north to avoid a possible continued buildup in the Souix Falls area.” 

He therefore sketched a more conservative northern deviation, checked the fuel consumption and 

said: 

“I probably opt for the computer-generated route, the orange one, knowing I could deviate 
further north and still make the completion of the trip.” 

‘Automatic’ - Subject 2. This dispatcher viewed the composite clouds and radar, saying: 

“There’s some good activity in there. I don’t see any real holes.” 

He looked at the forecast and concluded they: 

“can’t go over it.” 

He observed that the computer-suggested northern deviation: 

“Costs me a thousand pounds in fuel, about 8 minutes. Well within any sort of limitations 
I have for the airplane.” 

He looked at the winds and commented: 

“The winds to the south don’t look good,” 

and scrolled along the spreadsheet for the computer-suggested route, noting: 

“That’s a totally clean route according to the computer as far as turbulence and weather.” 
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He concluded: 


“I don’t see any reason not to do the computer-generated route.” 

Case 2 - Route Selection. In the previous study of 30 pilots, 6 of 10 using the ‘sketching only’ 
version went north and 4 went south; 9 of 10 using the ‘route constraints’ version went north and 
one went south; and all ten using the automatic version went north. The data for the 27 dispatchers 
are very similar. Of the 10 ‘sketching only’ subjects, 7 went north and 3 went south; of the 9 
‘route constraints’ subjects, 6 went north, 2 went south and one decided to weave through the 
storms. All 8 of the ‘automatic’ subjects deviated north. 

These results leave little doubt that, in spite of the fact that all of these subjects had potential access 
to the same data, the way in which the computer supported them in generating alternative routes 
had a very powerful influence on their situation assessments. 

Differences in Mental Models. There were very significant difference in the models of the weather 
developed by the different dispatchers: 

“Going south isn’t going to do anything for me but give me headwinds.” 

“In the summertime if this thing starts to build it will build faster to the south than to the 
north. Trying to go south could be a sucker hole. I don’t see a lot of weather potential to 
the north of the severe weather. A friend of mine got suckered that way and he went south 
and the line just beat him. It just built faster than he could get there. These sometimes can 
build up really fast and it went south on him and it just ate him alive. The airplane just flew 
forever. The reserves - there weren’t any. They ate a lot of reserves.” 
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“There should be very little further development south. If there is, it’s nothing that he can’t 
scoot by pretty quick.” 

An interesting hypothesis is that viewing the computer’s suggested northern deviation caused the 
subjects in the ‘route constraints’ and ‘automatic’ versions to look for justifications for such a 
northern deviation, thus changing their data collection process and their resultant mental models. 
This raises interesting possibilities that cognitive biases similar to a confirmation bias or biased 
assimulation are induced by viewing the computer’s suggestion (Fraser, Smith and Smith, 1992). 

Case 2 - Differences in Information Seeking. Table 2 indicates the numbers of dispatchers who 
looked at the different classes of map displays available. Of particular significance is the fact that 3 
of the 10 dispatchers in the ‘sketching only’ version never looked at the winds, one factor arguing 
against the southern route: 

“The more north we go the better winds we’re gonna get South we’re going to be in a 
headwind situation.” 

Table 2. Information Viewed in Case 2 



Radar 

Fronts 

Winds 

Clouds 

Jet Routes 

‘Sketching Only’ 

10/10 

8/10 

7/10 

3/10 

9/10 

‘Route Constraints’ 

9/9 

7/9 

6/9 

4/9 

7/9 

‘Automatic’ 

8/8 

4/8 

6/8 

5/8 

5/8 
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Equally interesting, 2 of 10 subjects using the ‘sketching only’ version did not look at the 
turbulence data on the spreadsheet for their southern routes. If they had done so, they would have 
discovered moderate turbulence along their planned deviation. 

Finally, it is interesting to note that a sizeable number of the dispatchers never looked at the 
forecast weather ( 7/10 in the ‘sketching only’, 6/9 in the ‘route constraints’ and 7/8 in the 
‘automatic’ version). Indeed, the dispatcher in the ‘route constraints’ version who chose a poor 
solution, letting the plane weave through the storm, was one of these subjects. In the debrief he 
noted his poor selection and blamed it on his failure to look at the forecast: 

“I forgot the most useful tool which is moving them [moving the displays forward in 
time].” 

(He also chose not to use the route constraints function and hence never saw the computer’s 
recommendation.) 

Case 2 - Other Interesting Behaviors. The data again showed clear examples of other 
considerations: 

“I tend to think in terms of where I’ve got support, where I can put this airplane if 
something goes wrong.” 

“This may be closer to the thunderstorm than I like, but again here we have a situation 
when the crew gets close, if they decide they want to deviate further they can.” 

In addition, some dispatchers used the system to work out in detail certain what-if situations: 
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“I’d select the orange route [the computer- suggested northern deviation] with the blue route 
[a more conservative northern deviation sketched by the dispatcher] as an alternate. 
Checking the fuel bums it’s only, the difference between the orange and the blue routes is 
only a difference of 3 minutes and 400 pounds. So I’d have him fly the orange route and 
give him the info to fly the blue route if that was necessary later.” 

As a last example, consider one dispatcher’s view of the job: 

“The orange route [the computer-suggested northern deviation] was the original reroute. 
That’s cool! I got it by 1 minute [with a route the dispatcher had just sketched]. This is 
what I do with my flight planning computer at work. I play chess with it [Then, after 
scrolling through the turbulence display on the spreadsheet for the sketched route:] Argh! 
Stabbed! Well, I beat it by 1 minute but it’s showing moderate turbulence on my route.” 

Case 2 - Summary. Like Case 1, the major results for the 27 dispatchers in Case 2 were very 
similar to those for the 30 pilots in the previous study. There were clearly problems with some 
dispatchers failing to look at important data at the right time; there were major differences in the 
mental models of the situation developed by different dispatchers; and the system version used had 
a very sizeable influence on the selection of an alternate route. Perhaps the most interesting insight 
inspired by this data is a possible explanation for the effect of viewing the computer-generated 
suggestion: 

Having seen the computer’s recommendation, some dispatchers may be influenced to look 
for data and explanations that justify the computer’s suggestion. This may lead to the 
development of a different mental model of the situation. 
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Case 3. Case 3 was designed to present the subjects with a difficult planning problem and to put 
the various system designs to a demanding test. Unlike the previous cases, the thunderstorms in 
Case 3 were not localized and their tops were not all at the same altitude. Like Case 2, there were 
two likely directions for deviating, but in this case neither was without potential problems. In 
particular, a deviation that avoided storms at the beginning of the route had to pass through more 
severe storms later. Finally, flight safety was a bigger concern on this case than the previous 
cases. 

Description of the Case. The following scenario was read to the subjects prior to their 
working on the case: 

“It’s summer and the plane is on a flight from Cheyenne to San Antonio. The plane got off 
the ground at 1900 Zulu and are now two minutes into the flight Decide what you think 
the plane should do.” 

The original route, the current aircraft position, and the current composite radar are shown in 
Figure 6. The current radar shows a number of thunderstorm cells with tops ranging from 28,000 
to 43,000 feet, but the aircraft’s maximum altitude was 33,000 feet. One of the cells directly on 
the flight path had a top of 43,000 feet. The forecast radar showed that the cells were predicted to 
move north and slightly east 

In summary. Case 3 presented subjects with a rather complex planning problem. The weather was 
dispersed over a large area and was changing somewhat unpredictably. This scenario required that 
the dispatchers anticipate various possible outcomes and plan accordingly. The routes suggested 
by the computer in the ‘route constraints and sketching’ and ‘automatic route .constraints, route 
constraints, and sketching’ conditions are shown in Figure 6. There were two routes suggested by 
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the computer, depending upon the constraints placed on it. Constraints of no turbulence and no 
precipitation caused the computer to suggest the eastern route (hereafter referred to as the ‘eastern’ 
route). Constraints that allowed light turbulence and precipitation caused the computer to suggest 
the western route (hereafter referred to as the ‘western’ route). In the ‘automatic route constraints, 
route constraints, and sketching’ treatment condition, the computer automatically suggested the 
eastern route to the subjects. These subjects had to modify the constraints on the computer or 
sketch their own route in order to come up with a western route. 


Insert Figures 6 and 7 about here 


The’ eastern route’ passed between two large, severe thunderstorm cells. Summer thunderstorms 
in Texas are notorious for their volatility and it was very possible that the two cells on either side of 
the ‘eastern route’ would grow and build together. Furthermore, the ‘eastern route’ passed 
extremely close to a forecast intense cell location. 

Case 3 - Hypotheses. In both Cases 1 and 2, the version of FPT used by pilots in our previous 
study and by dispatchers in this study had a sizeable influence on the cognitive processes involved 
in selecting an alternative flight plan, and on the route selected. Nevertheless, all of the pilots and 
dispatchers ultimately selected routes that were at least satisfactory for Cases 1 and 2. 

In Case 3, we selected a scenario (based on actual weather data provided by the National Center for 
Atmospheric Research) where the computer’s suggestion was very poor. 

Our previous study of the 30 pilots on Case 3 supported five hypotheses: 
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1 . Planners using the ‘route constraints’ and ‘automatic’ versions are much more 
likely to select the computer’s recommended solution which deviates to the east of 
the original flight path (a poor solution). This selection of a poor solution is in part 
due to a failure to consider the uncertainty associated with the weather forecast 
(something the computer similarly fails to consider); 

2. Because of the large “solution space”, planners using the ‘sketching only’ version 
are likely to often generate inefficient solutions (in terms of fuel and time); 

3 . Because of the large “data space”, planners may fail to note important data 
(turbulence information and radar forecasts) or become “disoriented”, failing to 
realize which weather display (current vs. forecast weather) they are looking at 

4. When a planner uses a short planning horizon to sketch a plan (making choices one 
segment at a time and not reconsidering previous choices), the result may be a 
sketched route similar to the computer’s suggested eastern deviation (the poor 
route). 

5 . Using an eliminations by aspects strategy, planners using the ‘route constraints’ 
and ‘automatic’ version may eliminate the ‘western route’ because of the predicted 
turbulence along the end of the flight and select the ‘eastern route’, even though in 
terms of a global evaluation it is clearly inferior. 

Case 3 - Sample Subjects. Three representative subjects (dispatchers) are described in detail 
below. 

‘Sketching Only’ - Subject 6. To begin, this subject looked at the curent and forecast weather, 
checking the composite radar and clouds and the fronts. He concluded: 
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“This stuff is pretty stationary, if anything drifting a little bit north. Some pretty hot cells, 
up to 43,000 right along the route. That’s the wrong place to be. This is a pretty bad 
bunch of weather here.” 

Hen then sketched a far western deviation from DEN to HBU to FMN to ABQ to CNX to ROW to 
INK to ABI to SAT. While doing so he commented: 

“I’m just gonna circumnavigate this whole area. It’s too nasty a weather system to be 
playing with. I’m gonna stay on the backside of this stuff.” 

He also noted that: 

“That route adds considerable time and bum, but it’s too crummy to get foolish here. I just 
don’t like that weather pattern. That thing is building and developing. This is definitely 
bad news. I juist have to eat it, add an extra half hour to the flight time.” 

Noting that he “could be cutting it a little bit fine here [in terms of fuel]” he then modified his far 
western route to fly more directly from INK to SAT (flying from INK to JCT to SAT instead of 
from INK to ABI to SAT). He checked the spreadsheet for turbulence along this new route, 
commenting: 

“It’s gonna hit some turbulence on the descent but there’s not a whole lot to do about that” 
In the end, however, he decided that: 

“Based on getting a smoother ride. I’d have to pick the slightly longer route [his initial far 
western deviation]”. 
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‘Route Contraints’ - Subject 16. This dispatcher looked at the fronts and weather radar for the 
current weather and then asked the computer to find a route with no turbulence or precipitation. 

The computer generated the ‘eastern route’ for which the dispatcher noted: 

“The fuel bum is up about 1700 pounds. We’re out of the turbulence and we’re out of the 
thunderstorm activity. ... I’d say that looks like a good route. ... The passengers will be 
comfortable and it looks like a good safe flight.” 

In the debrief, however, when he was shown both the computer-generated suggestions for 
avoiding the storm (one west of the storm and the other - which he had selected - east of the storm) 
and was shown all of the relevant weather data, he completely changed his opinion: 

“I should have gone with the other one [the computer-suggested western deviation]. I 
forgot about the forecast The fuel’s not that much different and it definitely keeps you 
away from all the thunderstorm activity. ...Please bum this tape.” 

‘Automatic’ - Subject 9. This dispatcher first looked at the composite radar for the current 
weather, commenting: 

“What a mess! The green [current] route stinks.” 

He then looked at the radar displays for the current weather at different altitudes. Stating that he 
wanted to “see where the weather is moving”, he displayed the composite radar and composite 
clouds and looked at how they changed from the current to forecast weather displays. His 
evaluation at this point was that: 

“It’s really hard to see movement Looks like the orange one [the computer’s suggested 
route that wound around through gaps in the storm to the east of the original route] is not 
perfect either.” 
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This dispatcher then sketched a route of his own that was west of the major storm activity, stating: 
“Let me see if there’s a way to get around it. Usually the weather moves from west to east, 
so maybe I could come around the back side.” 

He sketched a route from DEV to PUB to LVS to TCC to ROW to INK to JCT to SAT (see Figure 
7), commenting: 

“I wouldn’t want to be on this airplane, but we’ll try it.” 

The dispatcher then scrolled along the spreadsheet to look at the predicted turbulence and noted: 
“There’s turbulence on the descent” 

for this newly sketched route. He consequently raised the altitude for the last two legs of the flight 
(since only light chop was reported at this higher altitude, as opposed to moderate turbulence at 
lower altitudes). 

Finally, this dispatcher compared the two alternative routes under consideration (the computer 
suggested deviation east of the original route vs. his own route sketched west of the original route) 
and observed: 

“My route is not too much further. A little bit longer and a little less fuel [remaining at the 
destination], and they’re going to use more fuel to deviate at the end [to avoid or reduce the 
turbulence]. Personally, I think my pink route is a better because I’m behind the weather 
for most of the way. I’d take the pink route and let him [the pilot] deviate at the end.” 
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Case 3 - Route Selection. There was considerable variability in the plans selected by the 10 
dispatchers in the ‘sketching only’ version. One elected to land early at Amarillo; one made a 
minor deviation further east at the beginning of the flight (to avoid a perceived problem with the 
storm activity west of that part of the flight plan), and then followed the original route; three 
deviated further west than the ‘western route’ (one of these also made a jog east of the ‘western 
route’ at the end of the flight (see Figures 6 and 7); five sketched and selected the ‘western route’. 

Of the four subjects who deviated west of the ‘western route’, all agreed in the debrief (after being 
shown the ‘western route’ and all of the data) that this was an unnecessary extra deviation that was 
wasteful in terms of time and fuel. Like the results in Case 1, this illustrates the problems caused 
by the large “solution space” that has to be searched when sketching a plan. 

Two of the 10 ‘sketching only’ subjects did not look at the turbulence display on the spreadsheet, 
another illustration of the problems with having too large a “data space” (too many data displays to 
select from). One of these subjects commented in the debrief: 

“It really bothers me that I missed that moderate turbulence over there. Damm.” 

Of those that did look at the turbulence display, one chose to deviate further east for the last part of 
the flight (see Figure 7), while the others decided: 

“We’re going to have moderate turbulence all the way down through there. Should pose 
no problem for the aircraft other than a little bit of a bumpy ride”. 

and noted: 

“Sometimes there is no right answer. With this weather situation, if you can’t avoid the 
whole thing. You have to rely on the pilot to pick his way through.” 
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Of the 9 dispatchers using the ‘route constraints’ version, five selected the ‘western route’, one 
stayed on the original route, two chose the computer-suggested ‘eastern route’ and one slightly 
modified the coputer-suggested ‘eastern route’ to fly even further east at the beginning of the flight 
As with the study of 30 pilots, dispatchers who first generated the computer’s suggested ‘eastern 
route’ by selecting constraints of no turbulence and no precipitation were strongly influenced to 
select that route. Indeed, a finer grained analysis shows that, of the 10 pilots in the previous study 
who used the ‘route constraints’ version, 3 first generated the ‘eastern route’ using the route 
constraints function, and of these 3 two selected the ‘eastern route’. Similarly, of the 9 dispatchers 
in the ‘sketching only’ condition, 4 first generated the ‘eastern deviation’ using the route costrain ts 
function, and 3 of these selected the ‘eastern route’. 

It is interesting to also note that the dispatcher who modified the computer-suggested ‘eastern 
route’ to fly even further east at the beginning of the flight was influenced by looking at an 
inappropriate display. He should have been looking at the current radar weather. Instead (and 
apparently without being aware of it), he was looking at the forecast, which was for a time period 
where the plane would be well beyond the pont where he made his eastern deviation. This is an 
example of the “disorientation” described earlier. 

Finally, of the eight subjects using the ‘automatic’ version, all 8 deviated west of the storm (6 
followed the ‘western route’; 2 added a further deviation west at the beginning of the ‘western 
route’). 

It is interesting that all 8 of these subjects looked at the computer’s automatically suggested ‘eastern 
route’ and had immediate responses like: 
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“I don’t like going through what looks like, what the orange route [computer-suggested 
‘eastern route’] is doing, picked a hole in the front. Summer time this stuff can be pretty 
volatile. It’s liable to keep generating into a solid line.” 

“The route that the computer has selected is going to put the crew into the middle of a box. 
Once they arrive in this area they have no options.” 

They all consequently generated and selected a western deviation, even though they noted the 
turbulence along the end of that route: 

“Let’s keep it up at 330 until we get past Wink [to minimize exposure to the turbulence]. 
There’s not any way to avoid this stuff on descent down to San Antonio.” 

This contrasts with the performances of the 10 pilots in the previous study who used the 
‘automatic’ version. Four of those 10 subjects selected the computer’s suggested ‘eastern route’ 
even though they looked at an alternative’ western route’. (The ‘western route’ was rejected by 
those subjects because of the turbulence toward the end of the flight.) 

Case 3 - Differences in Mental Models and Situation Assessment.. As in Case 2, there were major 
differenc in the weather models and situation assessments by different dispatchers. All of them in 
the debrief concluded that the ‘eastern route’ was less desirable. Just how undesirable that route, 
and the original route were, however, varied widely. 

One dispatcher selected the original route and felt it was the best even in the debrief. This contrasts 
with assessments by other dispatchers like: 

“They’re showing tops at 43,000 which means we’re going to have to go around it” 
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“The green route [the original route] stinks.” 


Differences in evaluations of the ‘eastern route’ were even more interesting: 

“Probably gonna want to come around the back side of this stuff ‘cause its moving slowly 
east.” 

“I wouldn’t even attempt to take him around the east side because he’d have to go directly 
into that front activity and since that area’s moving west to east.” 

“I don’t like the easterly route because if I get pinched off and the line fills I really don’t 
have a good alternate choice other than going back up to Amarillo.” 

“Although it looks like there should be a pretty good hole, my general feeling would be that 
I would not trust that Down the road that far you could get caught in this line of 
thunderstorms, as it looks like it could very well form together.” 

“My thoughts remind me of the Southern crash, and I know just a little bit about that. That 
area is very susceptible, doesn’t give you much space in that particular area to go through 
those red, and what I call the red zone, and I have no desire to go through that and 
knowing that could fill in no time at all. That almost looks like that could be a front 
although its not indicated. Your ’re going to develop hail probably in that area and so forth 
which specifically could give you severe danger in that area.” 
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“That [the computer-suggested ‘eastern route’] is a pretty good route. I don’t have any 
problem with that” 

“There are four areas of severe thunderstorm activity. It’s what I’d still call scattered to 
widely scattered.” 

In addition, one dispatcher (in contrast to the other 26 dispatchers and 30 pilots), thought the 
weather was so bad he should just land: 

“Do I want to land at Amarillo and wait this out? Basing it strictly on safety, that would be 
my decision.” 

Case 3 - Differences in Information Seeking - As shown in Table 3 there were no clear 
differences in the information viewed by subjects using the three system versions. 

Table 3. In formation Viewed in Case 3 



Radar 

Fronts 

Winds 

Clouds 

Jet Routes 

‘Sketching Only’ 

10/10 

9/10 

8/10 

2/10 

10/10 

‘Route Constraints’ 

9/9 

8/9 

6/9 

4/9 

6/9 

‘Automatic’ 

8/8 

5/8 

5/8 

0/8 

8/8 


Case 3 - Summary. Many of the results in Case 3 were consistent with those from the previous 
study of 30 pilots. There was evidence that dispatchers in the ‘sketching only’ version had some 
difficulties with the large “solution space”, resulting in routes that were less efficient in terms of 
fuel and time, and there was evidence of “disorientation” where dispatchers were looking at the 
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forecast weather to plan the flight at points where the current weather display was more appropriate 
(or vice versa). 

Similarly, a significant number of dispatchers (3 of the 27) selected the clearly inferior ‘eastern 
route’ (compared to 8 of 30 pilots who did so in the previous study). The results further support 
the conclusion that, through various mechanisms, the computer’s suggested ‘eastern deviation’ 
influenced some dispatchers undesirably. 

In terms of this influence, it is interesting to note: 

1 . None of the dispatchers who saw both the ‘eastern’ and ‘western’ routes selected 
the poor ‘eastern route’. The three who did choose the poor route never explored 
the western alternative. This contrasts with many of the pilots who selected the 
computer-suggested ‘eastern route’ even though they looked at the ‘western route’; 

2. The data are suggestive that subjects in the ‘route constraints’ condition who 
viewed the ‘eastern route’ were more likely to select that route than subjects in the 
‘automatic’ version who viewed the same computer-generated suggestion; 

3 . Subjects using the ‘sketching only’ version were much less likely to select a poor 
eastern deviation (only 1 of 10 pilots and 0 of 10 dispatchers selected an eastern 
deviation). 

Case 4. Case 4 presented subjects with a situation in which the shortest and most fuel-efficient 
deviation, north, required the pilots to violate one of their standard heuristics (fly upwind of 
thunderstorms). The storm in this case could also be topped, although that would have put the 
plane in turbulence above the storm. Furthermore, there was some risk of the storm growing 
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quickly. As in the previous two cases, there were two likely directions for deviating; in this case 
those directions were north and south of the storm. 

The following scenario was read to the subjects prior to their working on the case: “The plane is 
on a flight from Albuquerque to New Orleans. It got off the ground at 1400 Zulu. It is now 19 
minutes into the flight and you’ve noticed a thunderstorm cell outside of Dallas. Decide what you 
think the plane should do.” 

The original route, the current aircraft position, and the current composite radar are shown in 
Figure 8 along with the likely deviations north and south of the storm. The forecast weather 
showed the storm moving slowly to the northeast. 

Case 4 - Hypotheses. Case 4 is primarily of interest as an opportunity to look at individual 
differences in situation assessment. The “solution space” is much smaller than for the first three 
cases, which is likely to reduce the impact of the effects of seeing the computer’s suggestion on 
plan selection. What is of interest is the fact that, as stated above, there is a conflict between 
different criteria for selecting a route. 

Case 4 - Route Selection.. There were three classes of solutions available, deviating north or south 
of the storm or staying on the original route (vectoring around the storm as necessary). Table 4 
shows the selections made by the 30 pilots in the previous study. Table 5 shows the results for the 
dispatchers in this study. 


Table 4. Route Selections bv Pilots 
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North 

South 

Original 

Sketching Only’ 

5/10 

4/10 

1/10 

Route Constraints’ 

5/10 

4/10 

1/10 

Automatic’ 

7/10 

2/10 

1/10 


Table 5. Route Selecti ons bv Dispatchers 



North 

South 

Original 

Sketching Only’ 

4/10 

5/10 

1/10 

Route Constraints’ 

4/9 

3/9 

2/9 

Automatic’ 

0/8 

7/8 

1/8 


The most interesting finding is again that there are sizeable individual dissferences in situation 
assessment and subsequent selection of a route. These differences will be explored in more detail 
in the next subsection. It is also interesting to note that the dispatchers using the ‘automatic” 
version had a much stronger preference for the southern deviation. 

Mental Models and Situation Assessment.. As the quotes below illustrate, there were very strong 
differences in the evaluation of the situation by different dispatchers. 

“Even though it looks like I can get over it, I think I would elect to look at some options 
here because I don’t know, this thing could develop and mushroom and continue to climb.” 
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“I would go with the pink route [northern deviation] at this time due to the computer telling 
me that there’s no turbulencce at that altitude, at 33,000, and the fuel consumption is pretty 
much negligible.” 

“I’d pick the northerly route. Even though the thunderstorm is heading in that direction, 
he’ll still avoid it and it is shorter than the southerly route.” 

“As far as fuel to destination and distance, the northern deviation would be shorter.” 
“North. There’s no point. The stuff is traveling north.” 

“Based on the movement of the convective activity I would probably recommend the 
southerly route.” 

“With a single cell moving at 1 5 knots, he can circumnavigate the cell. I see no reason to 
waste a thousand pounds of fuel for a deviation.” 

“I would tell him that turbulence has been reported in the vicinity of a severe thunderstorm, 
an isolated severe thunderstorm that’s in progress just west of DFW. If he went another 
20-30 miles off his course, he would probably minimize his fuel bum, get the same 
handling from ATC and minimize the delay on the flight He turns the seatbelt sign on, he 
starts a little bit to the south.” 

“Dallas traffic is not going to be a major consideration.” 
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“Coming down the southern route you’re gonna stay out of ATC’s way too.” 


Two points are worth noting: 

1. Only 3 of the dispatchers showed evidence of considering air traffic concerns. 

2. Although we are not in a position to determine which of the alternative solutions 
is best, they can’t all be. Hence, there is a need for better training to ensure that 
dispatchers correctly assess the weather situation and consider all of the relevant 
factors in selecting a route. 

Case 4 - Differences in Information Seeking - As shown in Table 6 there were no clear 
differences in the information viewed by subjects using the three system versions. 

Table 6. Inf ormation Viewed in Case_6 



Radar 

Fronts 

Winds 

Clouds 

.Tet Routes 

‘Sketching Only’ 

10/10 

8/10 

5/10 

3/10 

10/10 

‘Route Constraints’ 

9/9 

4/9 

6/9 

4/9 

6/9 

‘Automatic’ 

8/8 

5/8 

7/8 

5/8 

8/8 
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Conclusion 


This study, when combined with the results of our previous study of 30 airline pilots, leaves little 
doubt that the introduction of computer-generated suggestions for solving a flight planning 
problem can have a marked impact on the cognitive processes of the user and on the ultimate plan 
selected. In some cases, this impact is beneficial. If the computer’s model of the “world” is 
adequate for a particular scenario, the best route is more likely to be identified with the computer’s 
assistance. In other cases, however, the computer’s suggestion can have a profound adverse 
impact. When the computer makes a poor suggestion (because its model of the “world” is 
inadequate or because it doesn’t adequately consider all of the relevant factors), sizeable numbers 
of users are likely to be induced to accept this poor plan. 

The exact mechanisms by which such a negative influence is exerted merits further investigation. 
This study does, however, serve to identify some of the processes involved. First, some planners 
are overreliant in a very straightforward way: They accept the computer’s recommendation 
without seriously evaluating it. The performances of only 1 of the 7 pilots and 1 of the 3 
dispatchers who selected the poor ‘eastern route’ in Case 3 after seeing the computer’s suggestion 
can be accounted for by this explanation, though. The remainder selected a poor solution in spite 
of considerable efforts at evaluation. For these latter subjects, a major contributor to their failure 
appeared to be a failure to consider the limitations of the computer’s model of the “world”. They 
got drawn into the computer’s “world”, which did not take uncertainty in the weather into account, 
and consequently did not reason about the impact of this uncertainty on the desirability of the 
computer’s suggested “eastern route’ in Case 3. (The results in Cases 1 and 2 were also very 
consistent with this latter explanation, although the routes selected were much more satisfactory 
because the computer’s model of the “world” was adequate for those cases.) 
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In addition to these problems in coping with the large “solution spaces” that must be searched, and 
with the biasing effects of viewing the computer’s suggestion, there were problems with the large 
“data space” to be considered. Many of the dispatchers failed to view valuable information at 
appropriate times. System designers must therefore be very concerned with providing access to 
too much information, as this increases the chances that important data will be overlooked. The 
use of integrated data displays that combine all relevant data on a single display should also be 
seriously considered (the problem being that relevance depends on the situation), as should the 
inclusion of intelligent alerting functions to call attention to critically important data. 

General Design Concepts and Specific Implementation Details 

The biasing effect of the computer’s suggestion is problematic in suggesting design solutions. In 
our previous report on the study of 30 pilots we suggested that, instead of suggesting a single 
plan, expert systems techniques be introduced to allow the computer to suggest the best of each of 
the different classes of solutions available (leaving the choice among these alternatives up to the 
person). The assumption was that such a choice would influence the person to critically evaluate 
the alternative on a more global level. This suggestion still seems appealing after studying the 
dispatchers’ performances. It is critical, however, to recognize that the user is likely to be strongly 
biased to select from one of these computer-generated suggestions. Hence, it would be wise to 
embed an expert system on top of the optimization routine that filters out all but conservative 
suggestions for presentation to the user (thus putting the burden on the user to identify and evaluate 
plans that would in some situations be more risky). 
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The other general design concepts performed very favorably without any caveats, though. 
Whether we look at the subjective evaluations or the performance data, it is clear that the map 
display and the associated ability to sketch routes and view forecasts while moving the plane along 
its route are very desirable features, reducing the time to generate a route and allowing the user to 
explore “what-ifs”. Similarly, the ability to customize dispalys on the weather map was used 
effectively and viewed very positively by the dispatchers and pilots. Several specific 
improvements to the map display and associated functions were identified as part of this study, 
though: 


1. The system should support plans that involve vectoring as well as flights along 
jet routes; 

2. When the user is sketching a route along a jet route, the computer should highlight 
the jetroutes and navigational fixes directly accessable from the last point selected 
on the route. (Otherwise, users often have difficulty determining which fixes are 
connected by jetroutes.); 

3. One possible enhancement would be to allow users to skip a few navigational 

fixes when sketching a route (assuming the route is following jetroutes), allowing the 
computer to find a connecting path. One potential problem, though, is determining 
controlling how this route should be selected by the computer in terms of avoiding 
storm activity, etc. Another concern would be the possibility that the user would be 
less likely to critically evaluate such computer-generated flight segments; 

4. A function that allows the user to zoom in to variable levels of detail needs to be 
incorporated, as well as a means to easily move the area of focus north, south, east or 
west so that the user can easily look at adjoining areas on the map. Scroll bars 
along each edge of the map are likely to be an effective solution to meet this need; 
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5. Forecast maps representing weather every half hour or so for an extended period before 
and after the flight would be desirable (though not necessarily easy to create), so that 
the user could more easily assess trends in the weather. Caution is necessary in 
designing such a feature, though, as it may further increase the chances that the user will 
be drawn into the computer’s “world” which ignores uncertainty in the forecast (A 
means for enhancing the “world” to consider and display uncertainty is also an 
interesting design challenge that merits further consideration.); 

6. A major concern is the “disorientation” observed in some pilots and dispatchers, 

in which they unknowingly looked at the wrong weather (forecast instead of current or 
vice versa). Coding to accomplish this will be non-trivial, especially as forecasts for 
for more time periods are made available; 

7. The plane icon might be enhanced to show the plane’s altitude, etc. as a box within the 
icon, thus integrating this information into the map display; 

8. Additional weather and aircraft position data could be made available on the map 
display (jetstreams; turbulence; aircraft under the responsibility of that dispatcher; 
all aircraft in the area, etc.); 

9. A function could be provided to show all of the airports that the plane can reach for a 
diversion, or the airports that the plane can most quickly reach. This function might 
have the same type of “cognitive interface” that the route constraints function currently 
has, allowing the user to specifiy constraints on factors such as runway length, 
visibility, emergency and maintenance capabilies, passenger connections, etc.; 

10. The route constraints function could be enhanced to allow several other constraints 

to be set, such as searching only ATC preferred alternate routes, making at least X% of 
the passenger connections, etc. It could also be designed to allow the user to 
optimize time instead of fuel consumption; 


54 


Like the map display, the spreadsheet display fared very well in this evaluation. It was used often 
and effectively. It too could be enhanced, though, by allowing the user to embed additional 
graphics showing things like cloud tops. A design that allowed the user to choose between the 
current format and one that was scaled to distance and showed a continuous, exact altitude profile 
is also worth investigating. Finally, the ability to alter the altitude profile by direct manipulation of 
the graphic display would be desirable. 

One major area that needs further study has to do with workload concerns. At present, FPT 
supports planning for only one plane at a time. In reality, dispatchers are responsible for several 
planes at a time. Introducing this added complexity raises some fascinating issues regarding 
display design, and regarding the potential to adversely affect the user’s handling of the situation. 

Situation Assessment 

Two points merit discussion regarding this issue. First, it is clear that the three different versions 
of FPT studied here had major impacts on situation assessment, as discussed above. One 
interesting speculation regarding the cause of this effect is the suggestion that, having seen the 
computer’s suggestion, some dispatchers were influenced to look for data and explanations that 
accounted for the computer’s suggestion, thus changing the set of data viewed and the 
interpretation of this data. 

Second, this study makes it abundantly clear that dispatchers differ greatly in terms of the mental 
models that they develop give a particular set of weather data, and in terms of the factors they 
consider in evaluating the “goodness” of a particular route. This has major implications regarding 
the need for improved training. 
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Final Note 


It is clear that currently feasible technologies offer an opportunity to make significant 
improvements in the design of tools to aid in flight planning and airspace management These 
improvements can be realized in terms of increased safety, reduced costs, and more efficient and 
timely transport of passengers and cargo by air. 

It is also clear, though, that for the foreseeable future, these technologies will not be sufficiently 
powerful and reliable to fully automate complex tasks like flight planning. Consequently, we need 
to better understand how a computer can be designed to effectively work in cooperation with 
person, and how computers can be designed to enhance cooperation among different people. 

To gain such understanding, we need empirical studies, studies of performance in existing, real 
environments and more controlled studies in simulations of existing and more futuristic 
environments. We also need to recognize that the activity of developing futuristic environments 
like FPT is itself a form of research and a source of insights. Such prototypes provide 
representations to help us identify and explore important issues. In short, such prototypes, and the 
process of building them, help us to think about the issues more effectively. 
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